VGCN-BERT : augmenting BERT with graph embedding for text classification : application to offensive language detection

Le discours haineux est un problème sérieux sur les média sociaux. Dans ce mémoire, nous étudions le problème de détection automatique du langage haineux sur réseaux sociaux. Nous traitons ce problème comme un problème de classification de textes. La classification de textes a fait un grand progrès ces dernières années grâce aux techniques d’apprentissage profond. En particulier, les modèles utilisant un mécanisme d’attention tel que BERT se sont révélés capables de capturer les informations contextuelles contenues dans une phrase ou un texte. Cependant, leur capacité à saisir l’information globale sur le vocabulaire d’une langue dans une application spécifique est plus limitée. Récemment, un nouveau type de réseau de neurones, appelé Graph Convolutional Network (GCN), émerge. Il intègre les informations des voisins en manipulant un graphique global pour prendre en compte les informations globales, et il a obtenu de bons résultats dans de nombreuses tâches, y compris la classification de textes. Par conséquent, notre motivation dans ce mémoire est de concevoir une méthode qui peut combiner à la fois les avantages du modèle BERT, qui excelle en capturant des informations locales, et le modèle GCN, qui fournit les informations globale du langage. Néanmoins, le GCN traditionnel est un modèle d'apprentissage transductif, qui effectue une opération convolutionnelle sur un graphe composé d'éléments à traiter dans les tâches (c'est-à-dire un graphe de documents) et ne peut pas être appliqué à un nouveau document qui ne fait pas partie du graphe pendant l'entraînement. Dans ce mémoire, nous proposons d'abord un nouveau modèle GCN de vocabulaire (VGCN), qui transforme la convolution au niveau du document du modèle GCN traditionnel en convolution au niveau du mot en utilisant les co-occurrences de mots. En ce faisant, nous transformons le mode d'apprentissage transductif en mode inductif, qui peut être appliqué à un nouveau document. Ensuite, nous proposons le modèle Interactive-VGCN-BERT qui combine notre modèle VGCN avec BERT. Dans ce modèle, les informations locales captées par BERT sont combinées avec les informations globales captées par VGCN. De plus, les informations locales et les informations globales interagissent à travers différentes couches de BERT, ce qui leur permet d'influencer mutuellement et de construire ensemble une représentation finale pour la classification. Via ces interactions, les informations de langue globales peuvent aider à distinguer des mots ambigus ou à comprendre des expressions peu claires, améliorant ainsi les performances des tâches de classification de textes. Pour évaluer l'efficacité de notre modèle Interactive-VGCN-BERT, nous menons des expériences sur plusieurs ensembles de données de différents types -- non seulement sur le langage haineux, mais aussi sur la détection de grammaticalité et les commentaires sur les films. Les résultats expérimentaux montrent que le modèle Interactive-VGCN-BERT surpasse tous les autres modèles tels que Vanilla-VGCN-BERT, BERT, Bi-LSTM, MLP, GCN et ainsi de suite. En particulier, nous observons que VGCN peut effectivement fournir des informations utiles pour aider à comprendre un texte haiteux implicit quand il est intégré avec BERT, ce qui vérifie notre intuition au début de cette étude.Hate speech is a serious problem on social media. In this thesis, we investigate the problem of automatic detection of hate speech on social media. We cast it as a text classification problem. With the development of deep learning, text classification has made great progress in recent years. In particular, models using attention mechanism such as BERT have shown great capability of capturing the local contextual information within a sentence or document. Although local connections between words in the sentence can be captured, their ability of capturing certain application-dependent global information and long-range semantic dependency is limited. Recently, a new type of neural network, called the Graph Convolutional Network (GCN), has attracted much attention. It provides an effective mechanism to take into account the global information via the convolutional operation on a global graph and has achieved good results in many tasks including text classification. In this thesis, we propose a method that can combine both advantages of BERT model, which is excellent at exploiting the local information from a text, and the GCN model, which provides the application-dependent global language information. However, the traditional GCN is a transductive learning model, which performs a convolutional operation on a graph composed of task entities (i.e. documents graph) and cannot be applied directly to a new document. In this thesis, we first propose a novel Vocabulary GCN model (VGCN), which transforms the document-level convolution of the traditional GCN model to word-level convolution using a word graph created from word co-occurrences. In this way, we change the training method of GCN, from the transductive learning mode to the inductive learning mode, that can be applied to new documents. Secondly, we propose an Interactive-VGCN-BERT model that combines our VGCN model with BERT. In this model, local information including dependencies between words in a sentence, can be captured by BERT, while the global information reflecting the relations between words in a language (e.g. related words) can be captured by VGCN. In addition, local information and global information can interact through different layers of BERT, allowing them to influence mutually and to build together a final representation for classification. In so doing, the global language information can help distinguish ambiguous words or understand unclear expressions, thereby improving the performance of text classification tasks. To evaluate the effectiveness of our Interactive-VGCN-BERT model, we conduct experiments on several datasets of different types -- hate language detection, as well as movie review and grammaticality, and compare them with several state-of-the-art baseline models. Experimental results show that our Interactive-VGCN-BERT outperforms all other models such as Vanilla-VGCN-BERT, BERT, Bi-LSTM, MLP, GCN, and so on. In particular, we have found that VGCN can indeed help understand a text when it is integrated with BERT, confirming our intuition to combine the two mechanisms

Analysis of Influencing Factors of Tablet Consumer Satisfaction Based on Online Comment Mining

How to extract effective information that affects consumer satisfaction from online comments has become a hot issue for customer behavior. This article is based on the data mining of online comments and the research object are the top-selling tablets on the JD platform from October to December 2018. We started by analyzing influential factors such as goods, after-sales service, and logistics, and crawled online review information of nearly 3,000 tablet computers from five major brands. We first use the jieba word segmentation tool to process the user comments, and use TF-IDF to calculate the frequency of different words in the comments to determine the main keywords of the comments. Secondly, we set up a user\u27s sentiment dictionary to determine the sentiment index of the review, and combined the keywords and sentiment index to get the degree of consumer satisfaction on different influencing factors. Finally, we imported the quantified characteristic factors into Clementine 12.0, and established a Bayesian network model of customer satisfaction, thereby obtaining a ranking table of the importance of each factor to product sales. To improve the model robustness, we adopt a multivariate linear model to check the accuracy of the output results. Our research can not only formulate effective product service sales strategies for merchants, but also guarantee customers to experience better products and services

Research on Noise Reduction of Variable Speed Rotary Compressor with Large Capacity

With the increasing speed and capacity of variable-speed rotary compressors, the problem of noise especially low and medium frequency noise in the air conditioning system which can\u27t be solved by wrapping soundproof cotton has became more serious. In this paper, based on the noise problem of the rotor compressor with a working capacity of more than 80CC, the main frequency and the position of the noise source within 1000Hz are confirmed by simulation and experiment. Then on the base of this，the muffler and accumulator are respectively optimized and improved combining with Computer Aided Engineering (CAE) means. The final application results show that the optimized scheme can reduce noise by 6.1dB in 160Hz and 8.9dB in the frequency range of 500Hz to 800Hz, achieving good results

Investigation of a refrigeration system based on combined supercritical CO2 power and transcritical CO2 refrigeration cycles by waste heat recovery of engine

The majority of the energy in the fuel burned in the internal combustion engines is lost in the form of waste heat. To address this issue, waste heat recovery technology has been proposed to increases the overall efficiency of engine. This paper investigates a heat driven cooling system based on a supercritical CO2 (S-CO2) power cycle integrated with a transcritical CO2 (T-CO2) refrigeration cycle, aiming to provide an alternative to the vapour absorption cooling system. The combined system is proposed to produce cooling for food preservation on a refrigerated truck by waste heat recovery of engine. In this system, the S-CO2 absorbs heat from the exhaust gas and the generated power in the expander is used to drive the compressors in both S-CO2 power cycle and T-CO2 refrigeration cycle. Unlike the bulky vapour absorption cooling system, both power plant and vapour compression refrigerator can be scaled down to a few kilo Watts, opening the possibility for developing small-scale waste heat driven cooling system that can be widely applied for waste heat recovery from IC engines of truck, ship and trains.A new layout sharing a common cooler is also studied. The results suggest that the concept of S-CO2/T-CO2 combined cycle sharing a common cooler has comparable performance and it is thermodynamically feasible. The heat contained in exhaust gas is sufficient for the S-CO2/T-CO2 combined system to provide enough cooling for refrigerated truck cabinet whose surface area is more than 105 m2

Understanding electronic and optical properties of La and Mn co-doped anatase TiO2

AbstractThe electronic structures, dipole moment, and optical properties of La-doped, Mn-doped, and La-Mn-doped anatase TiO2 have been investigated by means of GGA + U first-principles method, showing that the absorption coefficients of the La-Mn-doped TiO2 under visible light to be sensitive to the doping positions of La and Mn atoms. La-Mn-TiO2(1) exhibits the largest absorption coefficient in visible light because of the smallest band gap. Additionally, for the La and Mn co-doped TiO2, the large dipole moment of TiO6 octahedron is in prejudice of enhancing the optical responses under the visible light. We successfully probe the interesting optical response mechanism in this work

Exploiting Field Dependencies for Learning on Categorical Data

Traditional approaches for learning on categorical data underexploit the dependencies between columns (\aka fields) in a dataset because they rely on the embedding of data points driven alone by the classification/regression loss. In contrast, we propose a novel method for learning on categorical data with the goal of exploiting dependencies between fields. Instead of modelling statistics of features globally (i.e., by the covariance matrix of features), we learn a global field dependency matrix that captures dependencies between fields and then we refine the global field dependency matrix at the instance-wise level with different weights (so-called local dependency modelling) w.r.t. each field to improve the modelling of the field dependencies. Our algorithm exploits the meta-learning paradigm, i.e., the dependency matrices are refined in the inner loop of the meta-learning algorithm without the use of labels, whereas the outer loop intertwines the updates of the embedding matrix (the matrix performing projection) and global dependency matrix in a supervised fashion (with the use of labels). Our method is simple yet it outperforms several state-of-the-art methods on six popular dataset benchmarks. Detailed ablation studies provide additional insights into our method.Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence (submitted June 2022, accepted July 2023